Real Cryptographic Protocol with an Insider Attacker: Improving Techniques for Proving Undecidability of Checking Security Goals

Existing undecidability proofs of checking secrecy of cryptographic protocols have the limitations ofnot considering protocols common in literature, which are in the form of communication sequences, sinceonly protocols as non-matching roles are considered, and not considering an attacker who is an insidersince only an outsider attacker is considered. Therefore the complexity of checking the realistic attacks,such as the attack to the public key Needham-Schroeder protocol, is unknown. The limitations have beenobserved independently and described similarly by Froschle in a recently published paper [1], where twoopen problems are posted. This paper investigates these limitations, and we present a generally applicableapproach by reductions with novel features from the reachability problem of 2-counter machines, and wesolve the two open problems. We also prove the undecidability of checking authentication which is thefirst detailed proof to the best of our knowledge. A unique feature of the proof is to directly addressthe secrecy and authentication goals as defined for the public key Needham-Schroeder protocol, whoseattack has motivated many researches of formal verification of security protocols. This report covers ourworkshop paper [2] and provide more details of modeling and proofs. ∗Research supported in part by NSF grants CCF 0306475 and DUE 0313880.